Increasing resolution and luminance of a display

ABSTRACT

The disclosed system modifies luminance of a display associated with a selective screen. The display provides a camera with an image having resolution higher than the resolution of the display by presenting multiple images while the selective screen enables light from different portions of the multiple images to reach the camera. The resulting luminance of the recorded image is lower than a combination of luminance values of the multiple images. The processor obtains a criterion indicating a property of the input image where image detail is unnecessary. The processor detects a region of the input image satisfying the criterion, and determines a region of the selective screen corresponding to the region of the input image. The processor increases the luminance of the display by disabling the region of the selective screen corresponding to the region of the input image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/537,908 filed Nov. 30, 2021, which claims priority to and the benefitof U.S. provisional patent application Ser. No. 63/283,902 filed Nov.29, 2021, both of which are incorporated herein by reference in theirentirety.

BACKGROUND

Large-scales displays such as light emitting diode (LED) walls can beused as backgrounds in filmmaking. However, large-scale displays tend tonot have very high resolution for several reasons. One reason is thatreducing the size of the light emitting diodes, to increase theresolution of the large-scale display, is difficult. In addition,bandwidth between the rendering engine and the large-scale displaylimits the resolution of the image that can be sent to the LED wall.

SUMMARY

Disclosed here is a system and method to increase resolution of adisplay, such as an LED wall. The display operates at a predeterminedfrequency by displaying a first image at a first time and a second imageat a second time. A selective screen disposed between the display andthe light receiver can include multiple light transmitting elements,such as pixel masks. A light transmitting element A can redirect a firstportion of light transmitted by the display. A light transmittingelement B can allow a second portion of light transmitted by the displayto reach the light receiver. The selective screen can increase theresolution of the display by operating at the predetermined frequencyand causing a first portion of the first image to be shown at the firsttime, and a second portion of the second image to be shown at the secondtime, where the first portion of the first image and the second portionof the second image are different. The predetermined frequency enablesthe light receiver to form an image based on the first portion of thefirst image, and the second portion of the second image.

In another implementation, the disclosed system and method can modifyluminance of a display. A processor can obtain an input image to presenton the display, where the display is associated with a selective screen.The display is configured to provide a light receiver with an imagehaving resolution higher than resolution of the display by presentingmultiple images associated with the input image while the selectivescreen enables light from different portions of the multiple images toreach the light receiver, thereby causing the light receiver to form theimage including the different portions of the multiple images. Theresulting luminance of the image is lower than a combination ofluminance values of the multiple images. The processor can obtain acriterion indicating a property of the input image where image detail isunnecessary. The processor can detect a region of the input imagesatisfying the criterion and determine a region of the selective screencorresponding to the region of the input image. The processor canincrease the luminance of the display by disabling the region of theselective screen corresponding to the region of the input image, therebydecreasing resolution of a region of the display corresponding to theregion of the input image.

In a third implementation, the disclosed system and method can increaseresolution of a display in postprocessing. The processor can obtainmultiple images presented on a display, where the display is configuredto present the multiple images at a first frame rate higher than a framerate needed to form a perception of motion. The processor can obtain amask corresponding to one or more images among the multiple images,where the mask indicates a portion of the one or more images among themultiple images to include in an output image. The processor canincrease resolution of the display in proportion to the number ofmultiple images presented to the display by combining, based on themask, the one or more images among the multiple images to obtain theoutput image.

BRIEF DESCRIPTION OF THE DRAWINGS

Detailed descriptions of implementations of the present invention willbe described and explained through the use of the accompanying drawings.

FIG. 1 illustrates an image represented by a set of pixels which defineminute areas of illumination on a display screen.

FIG. 2 illustrates a process for displaying, over a period of time, aseries of images representing a leaf portion on a pixel section of adisplay screen, at a first pixel resolution, which is then displayed ona pixel section of a selective screen, at a second resolution.

FIGS. 3A-3D illustrate a mechanical version for processing illuminationfrom a display screen via a selective screen.

FIG. 4 illustrates a section of a display screen and LEDs disposedbehind a section of a selective screen.

FIGS. 5A-5B show a top view of a display and a micro lens arrayconfigured to selectively allow light to reach a light receiver.

FIGS. 6A-6B show a position of a selective screen in a display stack.

FIG. 7 shows a darkening of an image recorded by a light receiver.

FIGS. 8A-8C show a light conductor according to various implementations.

FIGS. 9A-9B show various techniques to increase luminance of a display.

FIGS. 10A-10B show a virtual production set.

FIG. 11 is a flowchart of a method to increase apparent resolution of adisplay.

FIG. 12 is a flowchart of a method to modify luminance of a superresolution display.

FIG. 13 shows a system to increase resolution of an image inpostprocessing by displaying and recording multiple images at highfrequency.

FIGS. 14A-14C show various masks to use an increasing resolution of animage.

FIG. 15 shows a separation and different processing of a background anda foreground element.

FIG. 16 is a flowchart of a method to increase resolution of a displayin postprocessing.

FIG. 17 illustrates an example visual content generation system as mightbe used to generate imagery in the form of still images and/or videosequences of images.

FIG. 18 is a block diagram that illustrates a computer system upon whichthe computer systems of the systems described herein and/or an examplevisual content generation system may be implemented.

The technologies described herein will become more apparent to thoseskilled in the art from studying the Detailed Description in conjunctionwith the drawings. Embodiments or implementations describing aspects ofthe invention are illustrated by way of example, and the same referencescan indicate similar elements. While the drawings depict variousimplementations for the purpose of illustration, those skilled in theart will recognize that alternative implementations can be employedwithout departing from the principles of the present technologies.Accordingly, while specific implementations are shown in the drawings,the technology is amenable to various modifications.

DETAILED DESCRIPTION

Implementations provide a method and apparatus configured to display anoutput image from a display screen (e.g., monitor, television, LED wall,etc.) with a higher resolution image than the display screen is designedto natively produce (4K versus 2K resolution).

In one implementation, a display screen which may consist of multiplelight sources, such as light emitting diodes (LEDs), may be configuredto display a series of different lower resolution images configured toform, when combined, a high-resolution (“high-res”) image. The lowerresolution images may be displayed in some predefined order usingspecified locations or sections on the display screen which correlate tosections of the high-resolution image. Displaying the lower resolutionimages within a period of time, such as a time duration of a frame of avideo sequence, in the predefined order, fills in image details withinthe sections of the high-resolution image being recorded, eventuallyforming the high-resolution image.

To generate an image that is N times the resolution, a display may bedivided into N sections. N can be any integer greater than 1. Eachsection of the display is used to display a particular portion of theimage. The display receives a set of N number of predefined images ofthe image, which are then divided into multiple images. At least some ofthe multiple images associated with each section of the display areslightly different from each other in detail. The multiple images may bedisplayed in a predefined order, such as a sequential order, in arespective section of the display relative to their associated portionof the image within a specified time, such as the duration of a frame,to generate the image.

In some implementations, in order to display a higher resolution imagefrom a lower resolution image, a portion of one or more LEDs which forma pixel are reused over two or more video sequences to build imagedetail to form the image.

For example, in response to a first video signal associated with a firstimage, at a first time, the color of an LED (e.g., red) used may be setto the first color. A high-res selective screen with a pixel openingcorresponding to a first pixel of a high-res image may be placed infront of the LED to block light from all but a portion of the LED. Theportion of the LED not blocked provides a first pixel of the high-resimage at a first location with the first color. The resolution of theselective screen can be the same as the resolution of the high-resimage.

At a second time, the color of the LED may be set to a second color(e.g., green) in response to a second video sequence. The high-resselective screen may be placed in front of the LED image to block allbut a second portion of the LED, forming a second pixel of the high-resimage.

Thus, in this implementation, the multiple low-res images which areassociated with one or more sections and pixels of the high-resselective screen are employed to sequentially add detail to an imagerecorded by a light receiver.

Implementation Pertaining to Illumination

While illuminating an N number of sections of the display in some orderwill reduce the amount of display brightness generally by a factor of N,some implementations described herein may be used to offset thereduction of brightness. For example, larger LEDs may be used that areinherently brighter than the LEDs used for higher resolution displayssuch that the brightness of a lower resolution LED display may beconfigured with a brightness that is equal to or greater than the targetdisplay resolution.

In an illustration, a display of a certain fixed dimension withone-fourth the resolution may have LEDs that are four times larger andthus four times brighter than an LED display of the same fixeddimension.

Implementation Pertaining to Trading Between Brightness and Detail

In some implementations, image detail versus a desired brightness may beused to determine for a particular low-res display screen whether toemploy high-resolution processing as described herein, or whether to usea lower resolution image that is inherently brighter, such as the nativeresolution of the display screen.

For example, a view of a low detail background, such as a clear bluesky, may trigger a response to employ the native resolution of thedisplay screen to display the blue sky portion of the display imageinstead of the high-resolution image because, as in this example, moredetail would not be necessary to reproduce the blue sky background.

Mechanical Implementation—Moving Display Screen and/or Lenticular Lens

In other implementations, the selective screen and/or display screen maybe vibrated, e.g., moved in two or more directions, such that only a setof portions of pixels of the display screen is visible at a particularphysical position on the display screen. The selective screen and/or thedisplay screen may be vibrated using actuators or other mechanicalmechanisms as known in the art, such as mechanical actuators. Theselective screen includes at least two portions, one that allows lightto reach a light receiver, and one that redirects light. The portionthat redirects light can block the light from reaching the lightreceiver, can direct light away from the light receiver, or can focusthe incoming light to a portion of the light receiver.

In this implementation, the relative motion of the selective screen andthe display screen causes alignment between regions of the displayscreen and the region of the selective screen allowing transmission oflight from the display screen. When aligned with openings or lighttransports in the selective screen, only one or more pixels or portionsof the display screen transmit light through the selective screen, whilethe remaining portion of the pixels' light transmission is blocked bythe selective screen.

As the selective screen and/or the display screen is vibrated, a seriesof predefined images are displayed by the display at its nativeresolution in a particular order for each section over a predefined timeperiod, to produce a higher resolution image than what is native for thedisplay. In some implementations, the vibration or motion of the displayis synchronized with displaying the low-res images through transmissionsections of the selective screen to allow the sets of images to beprojected though the selective screen in a particular order.

Displaying the predefined images in the particular order, approximatelyin sync with the relative motion between the selective screen and thedisplay screen, allows one or more sections of the display to receiveand display images over a time period that results in producing thedesired level of detail for one or more portions of the image.

Thus, as the display screen and the selective screen move relative toeach other, the predefined images that pass through the selective screenor other light emitting apparatus, when visually combined in the eyes ofthe viewer, produce a resulting image with one or more portions of thedisplayed image having a higher resolution than the native resolution ofthe display screen.

Implementation Pertaining to Using a Grid of Liquid Crystal Displays toForm the High-Res Display

In some implementations, a set of liquid crystal displays (LCDs) areconfigured to form a high-res selective screen designed to be placed infront of a low-res display screen, such as an LED wall display.

Here, the LCDs of the high-res selective screen are sized and grouped ata resolution higher than the root resolution of the display. Theresolution of the LCDs can match the desired resolution of the high-resimage. For example, the LCD grid may be sized to display an image at 4Kfrom a display screen configured to natively display the image at 2K.

In one configuration, the LCDs are configured such that they can be setto transmissive state or to opaque state. In the transmissive state, oneor more LCDs allow a portion of light from a pixel of the low-resdisplay screen to pass through the selective screen forming higherresolution individual higher-res pixels of the selective screen. Inopaque state, one or more LCDs block or only partially allow the portionof light from a pixel of the low-res display screen to pass through thehigher-res pixels of the selective screen. In other implementations, theLCDs may be configured to amplify the light they receive.

Here, the low-res display screen is provided a set of images derivedfrom a high-res image where at least some of the set of images representdifferent higher-res pixels of the high-res image. When combined over aperiod of time, for example by image processing, the differenthigher-res pixels of the set of images provide image detail to form thehigh-res image. In one implementation, generating a high-res image fromthe set of images involves capturing a video sequence with a camera orother light receiver device.

For example, at a first video sequence time, the low-res displayreceives a first set of pixels representing a first image. The firstimage is displayed by the low-res display behind the selective screen.To display the hi-res version of the first image, a first set of pixelsof LCD are set to either transmissive state or opaque state. The LCDsset to transmissive state display the first set of high-res pixels forthe high-res image on the selective screen. The first set of LCDs set toopaque state fully or partially block the remaining light beingdisplayed on the low-res display screen. The camera captures the firstimage of the first set of high-res pixels from the selective screen.

At a second video sequence time, the low-res display receives a secondset of pixels representing a second image. The second image is displayedby the low-res display behind the selective screen. To display thehi-res version of the second image, a second set of pixels of theselective screen are set to either transmissive state or opaque state.The LCDs set to transmissive state display the second set of high-respixels for the high-res image on the selective screen. The second set ofLCDs set to opaque state fully or partially block the remaining lightbeing displayed on the low-res display screen. The camera captures thesecond image of the second set of high-res pixels from the selectivescreen.

In this implementation, the process continues until at least a thresholdnumber of the set of images have been presented via the selective screenin sequence and captured by the camera to form the resultant high-resimage either in the camera and/or using image processing to generate thehigh-res image which has a greater resolution than the display screencan natively produce.

Implementation Pertaining to Switching High-Res Mode on or Off Relativeto Camera View

In some implementations, to further increase brightness from theselective screen, the system may turn off or adjust high-res mode on oneor more sections of the selective screen that the camera is not imaging.For example, one or more sets of adjacent LCDs that are not in view ofthe camera may be set to transmissive state to allow more light from theunderlying LED of the low-res display screen to show through.

In this scenario, hi-res LEDs of the light transport display not in viewof the camera may be combined and employed to illuminate parts of thevirtual production set. For example, one or more sets of the lighttransport display not in view of the camera may be used for setlighting, shadows, patterns, effects, etc.

The description and associated drawings are illustrative examples andare not to be construed as limiting. This disclosure provides certaindetails for a thorough understanding and enabling description of theseexamples. One skilled in the relevant technology will understand,however, that the invention can be practiced without many of thesedetails. Likewise, one skilled in the relevant technology willunderstand that the invention can include well-known structures orfeatures that are not shown or described in detail, to avoidunnecessarily obscuring the descriptions of examples.

Increasing Apparent Resolution of a Display Using a Selective Screen

FIG. 1 illustrates an image 100 represented by a set of pixels whichdefine minute areas of illumination on a display screen (“display”) 110.In one scenario, different groupings of the set of pixels are parsedinto a set of images 100A-N that may contain different detailspertaining to image 100. The different groupings of pixels contain atleast some different details of image 100 that when combined produceimage 100. For example, consider image 100. Image 100A may be composedof even pixel columns of image 100, and image 100B may be composed ofodd pixel columns of image 100, such that when displayed or combined inan image processing system, generate image 100.

In some implementations, an image display system 102 includes displayscreen 110 at a first resolution, e.g., 2K pixels per inch (PPI), and aselective screen 120, having a second resolution which is typically at ahigher resolution, e.g., 4K PPI, than the display screen 110. Displayscreen 110 may be virtually any type of display screen that generatesimages through illuminating a grid of pixels.

In digital imaging, a pixel, pel, or picture element is a smallestaddressable element in a raster image, or the smallest addressableelement in an all points addressable display device. A pixel isconsidered the smallest controllable element of a picture represented onthe screen, such as display screen 110, selective screen 120, and thelike, and may be generated using light sources such as light emittingdiodes (LED) or lasers, may be formed from light projected onto ascreen, or may be formed by other means such as reflective surfaces,holes, light tubes, and the like. For example, display screen 110 may bederived from a group of LEDs formed into an electronic display such asan LED wall, a television, a computer monitor, a projection screen, etc.

In one configuration, selective screen 120 is positioned proximate todisplay screen 110 in a manner to receive illumination from displayscreen 110 on one side of selective screen 120, portions of which aretransmitted to an opposite side of selective screen 120 as describedherein. For example, selective screen 120 may include a set oftransmissive pixels configured to receive light from a light source,such as display screen 110, and transmit at least some wavelengths ofthe light source through the selective screen 120. In one configuration,the transmissive pixels may be formed using one or more grids of holes,apertures, slots, light tubes, and other types of structures, eitheractive or passive, that may be used to transmit, repeat, and/or amplifyat least some electromagnetic energy received on one side of selectivescreen 120, e.g., a light receiving side 120A, to another side ofselective screen 120, e.g., a display side 1208.

In one display process, images 100A-N may be presented in a particularorder on display screen 110. As images 100A-N are illuminated in theparticular order on display screen 110 using the first resolution,selective screen 120 receives illumination of images 100A-N on receivingside 120A, processes images 100A-N, and then transmits at least some ofthe received illumination through selective screen 120, which is thenemployed to illuminate display side 120B at the second resolution. Toform image 100, a light receiver 130 may capture images 100A-N in theparticular order displayed on selective screen 120, which may then becombined, for example, through an image processing system to form image100. The light receiver 130 can be an image receiving device, an imagecapture device, a camera, a video camera, a light receptor, an eye, etc.The light receiver 130 and the display 110 are synchronized in a mannercausing the light receiver to perceive the images 100A-N as a singleimage.

FIG. 2 illustrates a process for displaying, over a period of time, aseries of images 100A-N representing leaf portion 200 on a pixel section110A of display screen 110, at a first pixel resolution, which is thendisplayed on a pixel section 120E of selective screen 120, at a secondresolution. In one example process, selective screen 120 receivesillumination from section 110A at a first pixel resolution via receivingside 1208, employs selective screen 120 to process at least some of theillumination received at a second pixel resolution, displays images100A-N representing leaf portion 200 on display side 120C at the secondpixel resolution, and then employs light receiver 130 over the timeperiod to capture the series of images 100A-N at the second pixelresolution to form leaf portion 200.

In one example, portions of images 100A-N representing leaf portion 200are displayed at a first resolution in a particular order, over time,such as a frame, using LEDs 140 representing pixels and/or subpixels ofdisplay screen 110. Here, LEDs 140 of section 1108 of display screen110, when activated, illuminate a pixel section of receiving side 1208of the selective screen 120. The Illumination of the receiving side 1208of selective screen 120 is then processed and transmitted throughselective screen 120 to pixel display side 120C.

In some configurations, details of image 100 which are incapable ofbeing displayed on display screen 110 due to its lower native resolutionrelative to the native resolution of image 100 are formed via combiningimages 100A-N displayed by selective screen 120. For example, togenerate details of image 100 that are incapable of being displayednatively on display screen 110, in response to portions of images 100A-Nbeing displayed on section 110B, light from pixels 140 at a firstresolution, e.g., 2K, illuminates receiving side 120A of selectivescreen 120, and is then displayed via pixels 122 of selective screen120. Pixels 140 can include LEDs, LCDs, OLEDs, or any other passive oractive light transmitting element.

In one configuration, to display finer details from image 100, pixels122 of the selective screen 120 may be configured to be smaller in sizeand correspond to the desired resolution of the high-res image. Pixels122 may be positioned adjacent to pixels 140 in order to control theamount of light transmitted through selective screen 120. Because pixels140 illuminate corresponding pixels 122, when pixels 140 change colorover time in response to images 100A-N, pixels 122, configured totransmit light and/or direct at least some of the illumination, alsochange color.

In this example, at least some of pixels 122 may be positioned and/orcontrolled relative to adjacent pixels 140 to either transmit, partiallytransmit, redirect, or block light from such adjacent pixels 140 (e.g.,pixels 140 that pixels 122 can receive illumination from). Since pixels122 may be smaller than pixels 140, and are configured to allow only adesired portion of light to be transmitted and/or processed from pixels140 through selective screen 120, finer details of image 100 may bedisplayed via pixels 122.

For example, to form an edge 202 of leaf portion 200, pixelconfigurations 300A-D representing leaf portion 200 may be displayed onsection 110B of display screen 110 via LEDs 140. Section 110B of displayscreen 110 illuminates section 120D and 120E, which contains leaf edge202. In response, pixels 122 of section 120E are configured to allow orblock a portion of light from one or more pixels 140 that change to formpixel configurations 300A-D corresponding to at least some finer detailsof image 100 than displayed by display screen 110.

Here, at a first time, image 140A is displayed which illuminates LEDpixels 140 of display screen 110. At least some pixels 122 of section120E, here pixels 302, allow transmission of light from underlyingpixels 140A, while other pixels 122 of section 120E block light fromLEDs 140, forming pixel configuration 300A, corresponding to at leastsome finer details of image 100 than displayed by display screen 110.Light receiver 130 is then employed to capture the image created bypixel configuration 300A.

As can be seen in FIG. 2 , colors of the pixels 302 of the selectivescreen 120 correspond to the colors of underlying pixels 140A-140D.Pixels 120E of the selective screen are shown as covering a smaller areathan the pixels 140 of the underlying LED display for illustrationpurposes. In reality, pixels 120E of the selective screen overlap theunderlying pixels 140 of the LED display.

At a second time, image 1408 is displayed which illuminates LED pixels140 of the display screen 110. In this example, pixels 304 of section120E are configured to allow transmission of light from underlyingpixels 140B, while other pixels 122 of section 120E block light, whichforms pixel configuration 300B corresponding to at least some finerdetails of image 100 than displayed by display screen 110. Lightreceiver 130 is then employed to capture the image created by pixelconfiguration 300B.

At a third time, image 140C is displayed which illuminates LED pixels140 of the display screen 110.

In this example, pixels 306 of section 120E are configured to allowtransmission of light from underlying pixels 140C, while other pixels122 of section 120E block light, which forms pixel configuration 300Ccorresponding to at least some finer details of image 100 than displayedby display screen 110. Light receiver 130 is then employed to capturethe image created by pixel configuration 300C.

At a fourth time, image 140D is displayed which illuminates LED pixels140 of the display screen 110. In this example, pixels 308 of section120E are configured to allow transmission of light from underlyingpixels 140D, while other pixels 122 of section 120E block light, whichforms pixel configuration 300D corresponding to at least some finerdetails of image 100 than displayed by display screen 110. Lightreceiver 130 is then employed to capture the image created by pixelconfiguration 300D.

In this example, images captured of pixel configurations 300A-D are thencombined to form image 300E, a portion of the image 100. To combine thepixel configurations 300A-D, the light receiver 130 can be synchronizedto the display screen 110 and the selective screen 120 so that the lightreceiver 130 continuously records, e.g., keeps the shutter open, whilethe display screen 110 and the selective screen 120 display the full setof images needed to create the high-res, e.g., super resolution, image.

In the example of FIG. 2 , the resulting super resolution image has fourtimes the resolution of the underlying display screen 110 because eachpixel 210 (only one labeled for brevity) on the display screen 110 issubdivided into pixels 210A, 210B, 210C, 210D (only four labeled forbrevity) on the selective screen 120. The display screen 110 and theselective screen 120 are synchronized so that when the display screen110 is showing image 140A, the selective screen 120 allows light throughpixels 302 to pass. When the display screen 110 is showing image 1408,the selective screen 120 allows light through pixels 304 to pass. Whenthe display screen 110 is showing image 140C, the selective screen 120allows light through pixels 306 to pass, and when the display screen 110is showing image 140D, the selective screen 120 allows light throughpixels 308 to pass. The light receiver 130 is synchronized to thedisplay screen 110 and the selective screen 120 to continuously recordwhile the display screen and the selective screen cycle throughdisplaying pixel configurations 300A-D. Once all four images aredisplayed, the light receiver 130 closes the shutter, and records theimage containing light from all the pixel configurations 300A-D, toobtain the final image 300E.

FIGS. 3A-3D illustrate a mechanical version for processing illuminationfrom a display screen 110 via a selective screen 120. Here, pixels 122of a selective screen are positioned relative to pixels, e.g., LEDs 140,of display screen 110 such that certain portions of pixels of displayscreen 110 are either blocked or can transmit illumination throughselective screen 120.

Pixel 315 is a pixel on the display screen 110. In FIG. 3A, pixels 310,320, 330, 340 are pixels on the selective screen 120 placed in front ofthe pixel 315. Pixels 310, 320, 330, 340 cover portions 315A, 3158,315C, 315D of the pixel 315, respectively. Pixel 310 controls whetherthe camera 130 records the illumination from the upper-left portion 315Aof pixel 315. Pixel 320 controls whether the camera 130 records theillumination from the upper-right portion 315B of pixel 315. Pixel 330controls whether the camera 130 records the illumination from thelower-left portion 315C of pixel 315. Pixel 340 controls whether thecamera 130 records the illumination from the lower-right portion 315D ofpixel 315. If all pixels 310, 320, 330, 340 were to transmit light, thefull illumination of pixel 315 would pass through to the camera 130.

Pixel 310 can be configured to transmit light, and can be an aperture,such as a hole. When pixel 310 is positioned in front of differentportions of the underlying pixel 315, as shown in FIGS. 3A-3D, differentportions of the underlying pixel 310 are visible to the light receiver130.

In this example, the selective screen 120 may contain an illuminationblocking or redirecting structure 124 which is designed to separatepixels according to a pixel configuration of display screen 110. In oneimplementation, the blocking structure 124 can include pixels 320, 330,340, and can cover three-quarters of the surface area of pixel 315.Pixel blocking structures 124 can be positioned in front of every pixel315, 305 on the display screen 110. Consequently, blocking structures124 can allow only a portion of the illumination from pixels 140,including pixel 315, to reach the camera 130. The blocking structure 124and the pixel 310 form a portion 350 of the selective screen 120.

The blocking structure 124 can be an opaque material. The portion 350 ofthe selective screen 120 including the blocking structure 124 can beelectrochromic, so that the opacity of the blocking structure 124changes when voltage is applied to it. In one configuration, to allowlight to pass through pixel 320, and block pixel 310, voltage can beapplied to the blocking structure, so that the pixel 320 becomestransparent, while the pixels 310, 330, and 340 remain opaque.

The portion 350 of the selective screen 120 including the blockingstructure 124 can be a digital micromirror device. The digitalmicromirror device is a microscopically small mirror that can be laidout in a matrix on a semiconductor chip. These mirrors can be 5.4 μm orless in size. Each mirror represents one or more pixels in the projectedimage. The number of mirrors corresponds to the resolution of theprojected image. The mirror can be repositioned rapidly to reflect lighteither through the lens or onto a heat sink. The light that is reflectedthrough the lens can reach the light receiver 130, while the lightreflected into the heatsink does not reach the light receiver. Rapidlytoggling the mirror between these two orientations (essentially on andoff) produces grayscales, controlled by the ratio of on-time tooff-time.

A mirror among the multiple mirrors can correspond to the first lighttransmitting element among multiple light transmitting elements 510 inFIG. 5A. The mirror can be configured to be arranged in a first positionand a second position, where the first position of the mirror isconfigured to guide the first portion of light transmitted by the firstlight transmitting element toward the second portion of lighttransmitted by the first light transmitting element. The first positionof the mirror corresponds to the second light transmitting element thatredirects the incoming light. The second position of the mirror isconfigured to allow the second portion of light transmitted by the firstlight transmitting element among the multiple light transmitting element510 to reach the light receiver. The second position of the mirrorcorresponds to the third light transmitting element that allows incominglight to reach the light receiver.

In another configuration, selective screen 120 and/or display screen 110are moved via mechanical actuators and the like such that pixels 122 ofthe selective screen 120 and pixels 140 of the display screen 110 arepositioned relative to each other to allow a predefined portion ofillumination of LEDs 140 to pass through while blocking most, if notall, of the remaining illumination from LEDs 140. For example, at leastsome pixels 122 may provide a subpixel level of detail for LEDs 140.

Mechanical actuators 360, 370 can move portion 350 of the selectivescreen in horizontal direction 365 and vertical direction 375, thusenabling the pixel 310 to align with various portions 315A, 3158, 315C,315D of the pixel 315 and allowing light from the various portions ofthe pixel to reach the light receiver 130. As can be seen in FIGS.3A-3D, as the portion 350 of the selective screen 120 moves in thehorizontal direction 365 and in the vertical direction 375, fourdifferent areas of the underlying pixel 315 are exposed. When the camera130 records the four images, the resulting image 300E in FIG. 2 isproduced.

In one implementation, the mechanical actuators 360, 370 can move onlythe portion 350, covering the single pixel 315. In anotherimplementation, the mechanical actuators 360, 370 can move multipleportions 350, 355 of the selective screen 120 at the same time. Pixel315 can correspond to pixel 317 in FIG. 2 .

For example, at a first time shown in FIG. 3A, selective screen 120 anddisplay screen 110 may be positioned relative to each other such thatLEDs 140, when illuminated to display image 100A in FIG. 1 , causepixels 122 receiving the illumination to form an illuminated pixelconfiguration corresponding to pixel configuration 300A in FIG. 2 . Inthis case, pixel 310 allows light from pixel 315A to pass, while pixels320, 330, 340 block the light from pixels 315B, 315C, 315D,respectively. Light receiver 130 is then employed to capture the imagecreated by pixel configuration 300A.

At a second time shown in FIG. 3B, selective screen 120 can be movedhorizontally to the right to a second position such that LEDs 140, whenilluminated to display image 100B in FIG. 1 , cause pixels 122 receivingthe illumination to form an illuminated pixel configurationcorresponding to pixel configuration 300B in FIG. 2 . In this case,pixel 310 allows light from pixel 315B to pass, while pixel 330 blocksthe light from pixel 315D. Pixels 320, 340 block the light from aneighboring pixel 315 on the display screen 110. Light receiver 130 isthen employed to capture the image created by pixel configuration 300B.

At a third time shown in FIG. 3C, selective screen 120 can be moved downto a third position such that LEDs 140, when illuminated to displayimage 100C in FIG. 1 , cause pixels 122 receiving the illumination toform an illuminated pixel configuration corresponding to pixelconfiguration 300C in FIG. 2 . In this case, pixel 310 allows light frompixel 315C to pass, while pixel 320 blocks the light from pixel 315D.Pixels 330, 340 block the light from a neighboring pixel 315 on thedisplay screen 110. Light receiver 130 is then employed to capture theimage created by pixel configuration 300C.

At a fourth time shown in FIG. 3D, selective screen 120 can be moveddown and to the right to a fourth position such that LEDs 140, whenilluminated to display image 100D (not pictured in FIG. 1 ), causepixels 122 receiving the illumination to form an illuminated pixelconfiguration corresponding to pixel configuration 300D in FIG. 2 . Inthis case, pixel 310 allows light from pixel 315D to pass. Pixels 320,330, 340 block the light from a neighboring pixel 315 on the displayscreen 110. Light receiver 130 is then employed to capture the imagecreated by pixel configuration 300D.

In this example, as described herein, images captured of pixelconfigurations 300A-D may then be combined to form image 100, e.g., edge202 of leaf portion 200, displaying details which could not be displayedby display screen 110 at its native resolution.

FIG. 4 illustrates section 1108 of display screen 110 and LEDs 140disposed behind section 120E of selective screen 120. In thisconfiguration, display screen 110 and/or selective screen 120 may or maynot be moved, but rather may be configured as an electronic mask thatmay be employed to control what illumination is displayed from LEDs 140on display side 1208 of selective screen 120.

For illustration purposes, section 1108 is shown larger in area thansection 120E, while in reality the area of the section 1108 and the areaof the section 120E are the same. In FIG. 4 , each pixel in section 1108corresponds to four pixels in section 120E, however, othercorrespondences are possible such as each pixel in section 1108corresponds to 2, 3, 4, 5, etc., pixels in section 120E. Here, pixels122 may be formed from one or more selective screen sections 400.Selective screen sections 400 may be formed from any structure such as aliquid crystal display, or an electrochromic material, designed tocontrol the amount of illumination passing therethrough.

For example, referring to FIG. 2 , section 120E may include selectivescreen sections 400. The selective screen section can contain multipleregions. The region can be a pixel 410, or a group of pixels. The groupof pixels can be one or more columns 420, one or more rows 430, or anoncontiguous group of pixels 440. Each group of pixels can have aunique address, and can be controlled independently of the other groupsof pixels.

For example, to obtain the illumination pattern described in pixelconfigurations 300A-300D in FIG. 2 , the display screen 110 can beseparated into four noncontiguous groups of pixels 440, 450, 460, 470.Pixels belonging to group 440 can be the upper-left corner pixels,pixels belonging to group 450 can be the upper-right corner pixels,pixels belonging to group 460 can be the lower-left corner pixels, whilepixels belonging to group 470 can be the lower-right corner pixels. Inpixel configuration 300A in FIG. 2 , group 440 can allow the light topass, while groups 450, 460, 470 block the light. In pixel configuration300B in FIG. 2 , group 450 can allow the light to pass, while groups440, 460, 470 block the light. In pixel configuration 300C in FIG. 2 ,group 460 can allow the light to pass, while groups 440, 450, 470 blockthe light. In pixel configuration 300D in FIG. 2 , group 470 can allowthe light to pass, while groups 440, 450, 460 block the light.

As shown in FIG. 4 , column 420 forms a group of pixels that isindependently controlled from the rest of the pixels on the selectivescreen 120. As can be seen in FIG. 4 , column 420 allows light fromunderlying pixels on the display screen 110 to pass, while the rest ofthe columns block the light from the underlying pixels on the displayscreen 110.

FIGS. 5A-5B show a top view of a display and a micro lens arrayconfigured to selectively allow light to reach a light receiver. Themicro lens array 500 can be disposed between the light transmittingelements 510 of the display screen 110 in FIG. 1 , and a light receiver520. The light receiver 520 can be the light receiver 130, such as acamera, or a person's eye. The light transmitting elements 510 can bepassive or active light transmitting elements. Passive lighttransmitting elements can be liquid crystals in a liquid crystaldisplay, or digital micromirror devices. Active light transmittingelements can be light emitting diodes, organic light emitting diodes(OLEDs), etc.

The micro lens array 500 can include multiple lenses 530, 540 (only twolabeled for brevity). The lens 530, 540 can correspond to a single lighttransmitting element 515 among the multiple light transmitting elements510, or the lens can correspond to a column or a row of lighttransmitting elements. Each lens 530, 540 can include two portions 532,534 and 542, 544, respectively.

In FIG. 5A, portion 532, 542 is aligned with the first portion 515A ofthe first light transmitting element. The portion 532, 542 directs thelight away from the light receiver 520, thus preventing a portion 560 oflight transmitted by the first light transmitting element from reachingthe light receiver 520. The portion 534, 544 is aligned with the secondportion 515B of the first light transmitting element. The portion 534,544 directs the light toward the light receiver 520, thus allowing aportion 550 of light transmitted by the light transmitting elements 510to reach the light receiver 520. Consequently, the light receiver 520forms an image including light 550, but not light 560.

One or more actuators 570 can move the micro lens array 500 verticallyor horizontally. As seen in FIG. 5B, the micro lens array 500 has beenmoved horizontally by half the width of the single light transmittingelement 515. Once the micro lens array 500 has been moved, portion 534,544 is aligned with the first portion 515A of the first lighttransmitting element. Portion 534, 544 directs the light toward thelight receiver 520, thus allowing a portion 580 of light transmitted bythe light transmitting elements 510 to reach the light receiver 520. Theportion 532, 542 is aligned with the second portion 515B of the firstlight transmitting element. Portion 532, 542 directs the light away fromthe light receiver 520, thus preventing a portion 590 of lighttransmitted by the first light transmitting element from reaching thelight receiver 520. Consequently, the light receiver 520 forms an imageincluding light 580, but not light 590. As can be seen, differentportions of the light transmitting element 515 are visible to the lightreceiver 520 in FIGS. 5A and 5B.

The image as presented on the display 110 in FIG. 5A can be differentfrom the image presented in FIG. 5B. The light receiver 520 can keep theshutter open for the duration of the display of both images in FIGS. 5Aand 5B. Consequently, the light receiver 520 can form a resulting imageincluding a portion of both images in FIGS. 5A and 5B. The resultingimage has a higher resolution than the resolution of the display 110.

FIGS. 6A-6B show a position of a selective screen in a display stack.The selective screen 600 can be external to the display 110 stack asseen in FIG. 6A, or can be internal as seen in FIG. 6B. The display 110stack can include various layers depending on the type of the display.If the display 110 is an LCD display, the display 110 stack can includebacklight 610, an air gap 620, a diffuser 630, an LCD panel 640, and aprotective substrate 650. The selective screen 600 can be placed outsideof the display 110. If the display is an LED display, the display 110stack can include the LED panel 660, the selective screen 600 integratedinto the display 110 stack, and the protective substrate 670.

Increasing Luminance of a Time Multiplexed Super Resolution Display

FIG. 7 shows a darkening of an image recorded by a light receiver. Lighttransmitting element 700 at time T1 transmits illumination 705. Theselective screen element 710 allows only a portion of the illumination705 to reach a light receiver. The selective screen element 720 blocks aportion of the illumination 705 from reaching the light receiver. Thecombination of selective screen elements 710, 720 is drawn smaller thanthe light transmitting element 700 for illustration purposes only. Lighttransmitting element 700 at time T2 transmits illumination 730. At timeT2, the selective screen element 710 does not allow a portion of theillumination 730 to reach the light receiver, while the selective screenelement 720 allows a portion of the illumination 730 to reach the lightreceiver.

Light receiver element 740 is configured to record both the lighttransmitting element 700 at time T1 and the light transmitting element700 at time T2 to record the illumination 750A, 750B. Because the lightreceiver element 740 is recording the sum of illumination at both timeT1 and time T2, the recorded illumination 750A, 750B is darker than theillumination 705 and illumination 730, respectively. Specifically,illumination 750A is the sum of illumination 705 and 760, butillumination 750B is the sum of illumination 770 and 730. Consequently,in some cases, the disclosed system needs to increase the brightness ofthe display screen 110.

FIGS. 8A-8C show a light conductor according to various implementations.In FIG. 8A, display screen 110 and selective screen 120 are adjacent toeach other such that the illumination of LEDs 140 may illuminatereceiving side 120A of selective screen 120. Here, a portion ofselective screen 120, e.g., a portion of section 120E, is placed infront of a pixel 140 to block the view of pixel 140 from camera 130.

Here, in a predefined sequence, images 100A-N may be displayed bydisplay screen 110, which illuminates pixel(s) 140 of section 110E,which illuminates section 120E of selective screen 120 at receiving side120A. For example, with regard to FIG. 2 , at least some pixels 122 ofsection 120E, e.g., pixels 302, 304, 306, 308, allow or blockillumination transmission through selective screen 120, forming pixelconfigurations 300A-D in FIG. 2 , corresponding to at least some finerdetails of image 100 than displayed by display screen 110. In thisillustration, LCD 802A is configured to transmit or pass illumination804A, 804B, and LCD 802B is configured to block illumination 804A, 804B,from pixels 140.

In some configurations, to increase brightness from pixels 122, one ormore LCD sections may be configured with a light conductor 810, torefocus, channel, and or amplify energy blocked by LCDs used forblocking illumination from portions of pixels 140. The light conductor810 can be a light guide, a fiber optic cable, or a light pipe thatguides a light beam 804B using total internal reflection to a visibleportion of the pixel 802A. For example, if a brighter color is desired,a portion of the color being blocked may be channeled via lightconductor 810 and retransmitted as illumination 804A through pixel 122being used to form part of image 100.

Moreover, additional brightness may be obtained by channeling orredirecting light using the light conductor 810 from other adjacent LEDs140 that are currently being blocked, but that are not part of the setof pixels that make up one of the images 100A-N being displayed. Forexample, illumination from LEDs 140 outside the set of pixels 122 may beredirected between pixels using one or more light conductors 810 thatare configured to form one or more light channels between adjacentpixels 122. In this example, LEDs 140 outside the set of pixels 122 mayotherwise be illuminated to a desired color and their energy redirectedto one or more pixels 122. This configuration may be used to brightenthe overall resulting image 100 and/or brighten particular pixels 122.

In FIG. 8B, the light conductor 810 (only one labeled for brevity) canbe a micro lens 870, positioned above pixel 820 (only one labeled forbrevity) of the display 110. The micro lens 870 can move, thus focusingthe light coming from the pixel 820 to varying regions 830, 840, 850,860 of the light receiver 130 in FIG. 1 , as shown in FIG. 8C. Forexample, the micro lens 870 can be asymmetric, in other words nothemispherical. The apex of the micro lens 870A can be offset fromcenter. The rotation of the micro lens 870 can cause the micro lens tofocus the incoming light to region 850, as shown in FIG. 8C. An array880 of micro lenses can create a super resolution image by focusing theincoming light to the various regions 830, 840, 850, 860 of the lightreceiver 130.

FIGS. 9A-9B show various techniques to increase luminance of the display110. In FIG. 9A, a processor 900 can receive an input image 910 topresent on the display 110. To increase the brightness of certainregions of the input image 910, the processor can detect pixel 920 inthe input image 910 whose luminance is above a predetermined threshold.The predetermined threshold can be a weighted average of the minimum andmaximum luminance of the display 110, of the input image 910, of a lightreceiver 990, etc. If the luminance of the pixel 920 is above thethreshold, the processor 900 can disable the selective screen in frontof the pixel 920, thus reducing the resolution of the pixel 920 in theoutput image but increasing the brightness of the pixel 920 in theoutput image. Consequently, instead of the display showing an image 930with appropriate resolution and reduced luminance compared to the inputimage 910, the display shows an image 940 where pixels in the region 950are bright, but do not have the appropriate image detail 960. Forcertain regions of the input image 910, such as the sky, image detail960 may not be important. By contrast, regions 970, 980 presented on thedisplay 110 have the appropriate image detail but are darker compared tothe corresponding regions in the input image 910.

In FIG. 9B, the processor 900 can receive an indication 925 of an areaof interest 905 of the light receiver 990. The area of interest 905 canbe within the field of view of the light receiver 990, an area that isvisible to the light receiver 990 because it is within the field of viewand not blocked by props and/or actors in front of the display 110, oran area that is in focus for the light receiver 990. Generally, when aregion of the image is out of focus, higher luminance is desirablebecause the blur tends to spread the light of the out of focus region,thus reducing its luminance. Similarly, when the area is not visible tothe light receiver, super resolution of the area does not matter, andmore accurate luminance is desirable.

The region of the image that is inside the area of interest 905 can bepresented using the super resolution techniques described in thisapplication. The region 915 of the image that is outside the area ofinterest 905 can be presented at the native resolution of the display110. Consequently, the region of the image that is inside the area ofinterest 905 has high-resolution image detail, but low luminance, whilethe region 915 that is outside the area of interest 905 has lowresolution and low image detail, but high luminance.

The processor 900 can receive the indication 925 of the area of interest905 through a wired or a wireless network between the processor 900 andthe light receiver 990. For example, the processor 900 can have awireless transceiver in communication with a wireless transceiverassociated with the light receiver 990. In another implementation, theprocessor 900 can receive an image that the light receiver 990 hasrecorded in the previous frame. For the current frame, the processor 900can enable the selective screen in an area that is larger than theimage, while disabling the selective screen outside the area.

FIGS. 10A-10B show a virtual production set. The virtual production set1000 includes a virtual production display (“display”) 1010 which cansurround a stage 1020. Display 1010 can be the display 110. The display1010 can include multiple screens 1010A, 1010B, 1010C. One or more ofthe screens 1010A, 1010B, 1010C can be curved. The size of each screen1010A, 1010B, 1010C can correspond to a size of a wall and exceed 6 feetin height and 3 feet in width. The screen can use various displaytechnologies such as LCD, LED, OLED, rear projection, etc.

The stage 1020 is sufficiently large to include multiple actors 1030 andprops 1040, 1050, 1060. The stage 1020 can seamlessly integrate with thescreens 1010A, 1010B, 1010C presenting images 1015A, 1015B, 1015C,respectively. For example, the stage 1020 can include props, such asrocks 1050 and sand 1060, that mimic the appearance of rocks 1070 andsand 1080 that appear on the display 1010.

The display 1010 illuminates the stage 1020, actors 1030, and props1040, 1050, 1060. Thus, the lighting of the environment matches thelighting of the actors 1030 and props 1040, 1050, 1060. In particular,highly reflective surfaces, such as metallic surfaces, properly reflectthe environment. In addition to the display 1010 illumination,additional lights 1090 can illuminate the stage 1020.

The display 1010 needs to update the images 1015A, 1015B, 1015C toreflect events on the stage 1020 such as motion of the actors 1030,parallax to correctly create a sense of depth, interaction between theactors 1030 and the images 1015A-1015C, etc. In other words, the display1010 needs to render in real time. A rendering engine 1025, such asUnreal Engine or Gazebo, running on a processor 1035 can render theimages 1015A-1015C in real time in response to events on the stage 1020.The rendering engine 1025 can communicate with a camera 1005 using awired or a wireless network.

The camera 1005 can record the stage 1020 including images presented onthe display 1010, actors 1030, and props 1040, 1050, 1060. The camera1005 and the processor 1035 can each be coupled to a wirelesstransceiver 1065, 1045, respectively, through which the rendering engine1025 can track the camera movement, and through which the processor andthe camera can communicate.

FIG. 11 is a flowchart of a method to increase apparent resolution of adisplay. In step 1100, a software or hardware processor executinginstructions described in this application can operate a display at apredetermined frequency by causing the display to present a first imageat a first time and a second image at a second time. The predeterminedfrequency can be anywhere between 12 and 100 frames per second (FPS),such as 24 FPS, 30 FPS, 60 FPS, etc. The display can include a first setof light transmitting elements defining the resolution of the display.For example, if the display is an LED display, the first set of lighttransmitting elements includes LEDs. If the display is an LCD display,the first set of light transmitting elements includes liquid crystals.If the display is an OLED display, the first set of light transmittingelements can include organic light emitting diodes. If the display is arear projector or a front projector, the first set of light transmittingelements can include digital micromirror devices. The selective screencan also act as the reflector to reflect light coming from the projectortowards the display.

The frequency associated with the light receiver indicates a length oftime the light receiver needs to form the image. If the light receiver'sfrequency cannot be adjusted, the predetermined frequency corresponds toa frequency associated with the light receiver divided by at least anumber of elements in the multiple light transmitting elements. Forexample, the predetermined frequency can be equal to the frequencyassociated with the light receiver divided by the number of elements inthe multiple light transmitting elements.

In step 1110, the processor can increase the resolution of the displayby operating a selective screen at the predetermined frequency andcausing a first portion of the first image to be shown at the firsttime, and a second portion of the second image to be shown at the secondtime. The first portion of the first image and the second portion of thesecond image can be different. The predetermined frequency enables thelight receiver to form the image based on the first portion of the firstimage, and the second portion of the second image. The selective screencan include a second set of light transmitting elements defining theresolution of the selective screen. The second set of light transmittingelements can include a micro lens, a digital micromirror device, aliquid crystal display, or an electrochromic layer. The selective screenis disposed between the first set of light transmitting elements and thelight receiver. The light receiver can be a camera or a person's eye. Afirst light transmitting element among the first set of lighttransmitting elements can correspond to multiple light transmittingelements associated with the selective screen. The resolution of theselective screen can be higher than the resolution of the display. Theresolution is defined as the number of light transmitting elements perunit length, such as inch.

In step 1120, the processor can cause a second light transmittingelement among the multiple light transmitting elements to redirect lighttransmitted by the first light transmitting element. In step 1130, theprocessor can cause a third light transmitting element among themultiple light transmitting elements to allow light transmitted by thefirst light transmitting element to reach the light receiver. The secondlight transmitting element and the third light transmitting element canform at least a part of a liquid crystal display, a digital micromirrordevice, or an electrochromic display.

The second light transmitting element and the third light transmittingelement can each have a unique identifier. Alternatively, the secondlight transmitting element can be part of a first group of pixels, wherethe whole first group has a unique identifier. Similarly, the thirdlight transmitting element can be part of a second group of pixels,where the whole second group has a unique identifier different from theidentifier associated with the first group.

The processor can obtain a second unique identifier associated with thesecond light transmitting element, and a third unique identifierassociated with the third light transmitting element. Based on thesecond unique identifier, the processor can instruct the second lighttransmitting element to change a state. The state of the second lighttransmitting element can include an opaque state configured to blocklight transmitted by the first light transmitting element, and atransmissive state configured to allow light transmitted by the firstlight transmitting element to pass. Based on the third uniqueidentifier, the processor can instruct the third light transmittingelement to change a state of the third light transmitting element.Similarly, the state of the third light transmitting element can includethe opaque state configured to block light transmitted by the firstlight transmitting element, and the transmissive state configured toallow light transmitted by the first light transmitting element to pass.The processor's instructing the second light transmitting element andthe third light transmitting element occurs at the predeterminedfrequency. The instructions sent to the second and third lighttransmitting elements are synchronized to the predetermined frequency ofthe display. In other words, when the display changes, the state of thesecond and third light transmitting elements changes.

The processor can enable light from a first portion of a first imageamong the multiple images to reach the light receiver when the displayis presenting the first image. The processor can enable light from asecond portion of a second image among the multiple images to reach thelight receiver when the display is presenting the second image. Thefirst portion of the first image and the second portion of the secondimage occupy different regions of the display. To enable the light toreach the light receiver, the processor can change a state of at least aportion of the selective screen at the predetermined frequency. Changingthe state can include moving the at least a portion of the selectivescreen or changing transmission properties of the at least a portion ofthe selective screen. The transmission properties can include opacity,polarization, and/or lens alignment. The processor can align the portionof the selective screen with the first portion of the first image whenthe display is presenting the first image, and align the portion of theselective screen with the second portion of the second image when thedisplay is presenting the second image. The portion of the selectivescreen can allow light from the display to reach the light receiver.

The processor can operate the light receiver at a second predeterminedfrequency, where the second predetermined frequency is lower than afrequency of operation of the display. The second predeterminedfrequency can be equal to the predetermined frequency divided by anumber of elements in the multiple light transmitting elements.

The processor can cause a light conductor associated with a firstportion of the selective screen to guide a light beam blocked by thefirst portion of the selective screen to a second portion of theselective screen, thereby increasing brightness of light emanating fromthe display. The first portion of the selective screen can redirectlight from the display, and the second portion of the selective screencan allow light from the display to reach the light receiver. The lightconductor can be a light guide, a fiber optic cable, or a light pipe.

FIG. 12 is a flowchart of a method to modify luminance of a superresolution display. In step 1200, a processor can obtain an input imageto present on the display. The display can be associated with aselective screen, as described in this application. The display can be avirtual production display as described in FIG. 10 . The display isconfigured to provide a light receiver with an image having resolutionhigher than the resolution of the display. To achieve the highresolution, the display can present multiple images associated with theinput image while the selective screen enables light from differentportions of the multiple images to reach the light receiver. Themultiple images can be varying portions of the input image, such thatthe multiple images put together create the input image. One of themultiple images can be the input image, and the other multiple imagescan correspond to other input images. By recording the multiple imagespresented by the display, the light receiver forms a super resolutionoutput image including the different portions of the multiple images.While the output image has higher resolution than the input image, theluminance of the output image is lower than a combination, e.g., a sum,of luminance values of the multiple images. The camera, the selectivescreen, and the display are synchronized in a manner causing the camerato perceive the multiple images as a single image.

In step 1210, the processor can obtain a criterion indicating a propertyof the input image where image detail is unnecessary. In step 1220, theprocessor can detect a region of the input image satisfying thecriterion. The region can be a single pixel or a group of pixels.

In step 1230, the processor can determine a region of the selectivescreen corresponding to the region of the input image. In step 1240, theprocessor can increase luminance of the display by disabling the regionof the selective screen corresponding to the region of the input image,thereby decreasing resolution of a region of the display correspondingto the region of the input image.

In one implementation, to obtain the criterion, the processor can obtaina threshold between a minimum luminance of the display and a maximumluminance of the display. To detect the region, such as a pixel or agroup of pixels, the processor can detect the region of the input imagehaving an original luminance above the threshold. The processor candetect an edge in the image. The processor can determine whether theregion of the input image is proximate to the edge in the image. Upondetermining that the region of the input image is proximate to the edgein the image, the processor can enable the region of the selectivescreen corresponding to the region of the input image. Upon determiningthat the region of the input image is not proximate to the edge in theimage, the processor can disable the region of the selective screencorresponding to the region of the input image. In other words, if theright pixel is close to an edge in the image, the processor preservesthe super resolution of the pixel because edge regions are important tohuman vision.

In another implementation, the processor does not have to perform edgedetection, and upon detecting a region whose luminance is above thethreshold, the processor can disable the selective screen in front ofthe region.

In a third implementation, to obtain the criterion, the processor canobtain a region of focus associated with the light receiver. To detectthe region, the processor can detect the region of the input imageoutside the region of focus associated with the light receiver.Consequently, the processor can increase brightness and decreaseresolution of the region that is not in focus of the light receiver. Toobtain the region of focus, the processor can receive from the lightreceiver an indication of lens settings of the light receiver and lightreceiver position.

In a fourth implementation, to obtain the criterion, the processor canobtain an indication to detect an area associated with the displayinvisible to the light receiver, or an area that is not directly visibleto the light receiver. The invisible area, or an area that is notdirectly visible to the light receiver, can be outside the camera'sfield of view, or the area can be obscured by actors or props in frontof the display. The processor can detect the region of the input imageinvisible to the light receiver, and can disable the selective screen infront of the region.

In a fifth implementation, to obtain the criterion, the processor canobtain an input from a user indicating in which area of the displayscreen to trade-off luminance for detail. For example, the processor canpresent a slider to the user to indicate an amount of trade-off betweenand detail.

The processor can divide the region of the selective screen into a firstregion and a second region, where the second region provides atransition region between the first region and a remainder of thedisplay. For example, the second region can surround the first region sothat the second region borders the remainder of display and the firstregion, while the first region only borders the second region. Theprocessor can obtain a first luminance associated with the first region,and a third luminance associated with the remainder of the display. Theprocessor can adjust a second luminance associated with the secondregion according to an interpolation between the third luminance and thefirst luminance. To adjust the luminance, the processor can adjust thebrightness of the screen or can adjust the transparency of the selectivescreen.

Increasing Resolution of a Display in Postprocessing

FIG. 13 shows a system to increase resolution of an image inpostprocessing by displaying and recording multiple images at highfrequency. Display 110, such as virtual production display 1010 in FIG.10 , can present multiple images 1300A, 13008, 1300C at high frequency.The high frequency can be a frame rate higher than a frame rate neededto form a perception of motion. For example, a human eye can perceivemotion if presented with 12 images per second. Most film cameras operateat a base frame rate such as 24 FPS, but other frame rates are possiblesuch as 48 FPS, 30 Hz, 60 Hz, etc. The display 110 can operate at aframe rate that is the base frame rate times the number of images, e.g.,subframes, 1300A, 1300B, 1300C displayed within a base rate singleframe. For example, if the display 110 base rate was 24 FPS, but thedisplay 110 was displaying four subframes in place of one base ratesingle frame, the display 110 can operate at 4×24=96 FPS.

The light receiver 1310, such as a camera, can operate at a second framerate proportional to the frame rate of the display 110. If the framerate of the display 110 increases, the frame rate of the light receiver1310 also increases.

For example, if the light receiver 1310 operates at 24 FPS, the lightreceiver shutter usually captures light for a 48th of a second, then isoff for a 48th of a second. Let us assume that we want to capture atleast four subframes 1300A, 13008, 1300C, 1300D (not pictured) for eachbase rate single frame (“frame”), however other numbers of subframes arepossible such as 2, 3, 5, 6, etc. To accomplish this, the light receivercan operate at 192 FPS, taking eight subframes per frame with no “off”gap between them. Then the light receiver 1310 can discard the fourframes that correspond to the “off” period of the exposure. In analternative implementation, the light receiver 1310 can operate at 96FPS, taking a burst of four subframes at 1/192 of a second apart, thenhave a gap for a 48th of a second. In this implementation, the lightreceiver 1310 does not store unnecessary pictures on disks, such as thediscarded four frames, and does not overheat the electronics makingthem.

In a third implementation, the light receiver can operate at 96 FPS,taking pictures continuously. The third implementation produces a higherquality image because the subframes have 96th of a second exposure, not1/192 of a second exposure, so less light is required on set to record aclean image at the light receiver 1310.

The light receiver 1310 can record the images 1300A, 1300B, 1300C toobtain recorded images 1320A, 1320B, 1320C. A processor associated withthe light receiver 1310 can combine the recorded images 1320A, 1320B,1320C to obtain an output image 1340. The output image 1340 can be thehigh-res image.

To combine the recorded images 1320A, 1320B, 1320C, in oneimplementation, the processor can apply masks 1330A, 1330B, 1330C toeach of the recorded images. Each mask 1330A, 1330B, 1330C blocks off aportion of the corresponding image 1320A, 1320B, 1320C and allows aportion of the corresponding image to be visible. When the three masks1330A, 1330B, 1330C are applied to a single image in turn, the outputimage is the single image.

To combine the recorded images 1320A, 1320B, 1320C, in a secondimplementation, the processor does not rely on the recorded images1320A, 1320B, 1320C, and instead determines a region of the multipleimages visible to a light receiver based on the image as presented onthe display 110. In this implementation, even if the camera is moving,and different portions of the display 110 are visible to the lightreceiver 1310, the processor can combine pixels from various regions ofthe display 110 to obtain a pixel in the output image 1340. By combiningpixels from various parts of the subframes 1300A, 1300B, 1300C, theprocessor can create an appearance of motion blur.

In the second implementation, the processor can obtain images presentedon the display 110 via metadata. In one implementation, the metadata caninclude the images presented on the display 110. In anotherimplementation, the metadata can include a timecode that synchronizesimages presented on the display 110 and those captured by the lightreceiver 1310. The timecode keeps different cameras on set and displayscreens 1010A, 1010B, 1010C synchronized, and also gives each frame aunique number, consistent across all devices. Once the processor knowsthe timecode such as 1, 2, 3, 4, the processor knows which image ispresented on the display 110.

To determine which pixel in the output image 1340 corresponds to whichpixel on the display 110, the processor can obtain the display 110position and light receiver 1310 position in each frame. The informationabout position can come from a set survey and/or a motion capture deviceon the light receiver 1310. The motion capture device can be operatedwith a motion control system. Based on the light receiver 1310 positionper subframe, and the display 110 position, the processor can determinea relationship between pixels in the recorded images 1320A, 1320B, 1320Ccaptured by the light receiver 1310 and the pixels in the images 1300A,1300B, 1300C on the display 110.

To determine a relationship between pixels 1350A, 1350B, 1350C in therecorded images 1320A, 1320B, 1320C and the pixels in the presentedimages 1300A, 1300B, 1300C, each pixel 1350A, 1350B, 1350C in therecorded image is taken in turn. That pixel 1350A, 1350B, 1350Ccorresponds to a ray 1370 in space. A ray/plane intersection testproduces the precise position of a pixel 1360A, 1360B, 1360C on thedisplay 110 that corresponds to the pixel 1350A, 1350B, 1350C. Thepixels 1350A, 1350B, 1350C can be the same pixel in the light receiver1310. The pixels 1360A, 1360B, 1360C can be the same pixel, or can bedifferent pixels due to the motion of the light receiver. The color ofthe pixels 1360A, 1360B, 1360C comes from whichever subframe ispresented on the display at the time when the image 1320A, 1320B, 1320Cwas recorded.

For example, let us assume that there is no motion blur and there arefour masks 1330A, 1330B, 1330C, 1330D (not pictured) each of whichblocks one fourth of a pixel in the corresponding image. The pixel 645,455 in the light receiver's recorded image 1320A, 1320B, 1320C mightcorrespond to the upper-right quadrant of pixel 200, 205 in the display110. Because the subframe sequence is known, the processor knows theupper-right quadrant of pixel 200, 205 was displaying subframe 2 out of4, so pixel 645, 455 in the output image comes from pixel 645, 455 inthe second subframe.

When there is motion blur, a pixel in the output image corresponds to aline across the screen, as the light receiver 1310 movement causes therecording pixels of the light receiver to trace a path across thedisplay 110. To handle motion blur, the output value of the pixel 1350A,1350B, 1350C may have to come from multiple subframes fused together,possibly from slightly different pixel locations.

FIGS. 14A-14C show various masks to use in increasing resolution of animage. Each mask 1400, 1410, 1420, 1430, 1440, 1450, 1460, 1470, 1480can be a pixel mask, meaning that each mask 1400-1480 can be applied toa single pixel. Masks 1400, 1410 are complementary masks that can beapplied to two subframes, thus increasing the resolution of an image by2. Masks 1420, 1430, 1440 are complementary masks that can be applied tothree subframes, thus increasing the resolution of the output image by3. Finally, masks 1450, 1460, 1470, 1480 are complementary masks thatcan be applied to four subframes, thus increasing the resolution of theoutput image by 4. Masks increasing the output of the resolution imageby 5, 6, 7, etc., are also possible.

FIG. 15 shows a separation and different processing of a background anda foreground element. The display 110, such as the virtual productiondisplay 1010 in FIG. 10 , can have various foreground elements 1500,1510, such as actors and props, placed in front of the display 110.Foreground elements 1500, 1510 are not limited to the resolution of thedisplay 110 because they are not presented on display 110, and thusincreasing the resolution by combining multiple images using masks isunnecessary. Recording the foreground elements at a high frequency, suchas 96 FPS, can cause a strobing effect, and thus the foreground elementsmay need to be treated differently to create a sense of motion blur.

To detect foreground elements 1500, 1510, a processor can compare pixels1350A, 1350B, 1350C in the recorded images 1320A, 1320B, 1320C in FIG.13 , as recorded by the light receiver 1310 in FIG. 13 , to thecorresponding pixels 1360A, 13608, 1360C on the display 110. If the twocolors are different from each other above a predetermined threshold,then the processor concludes that the pixels belong to a foregroundelement 1500, 1510.

The processor can then separate the foreground elements 1500, 1510 fromthe background 1520, and apply different processes to each. For example,for the background 1520, multiple subframes 1530A, 1530B, 1530C can becombined with their corresponding masks 1540A, 1540B, 1540C to obtain asuper resolution output image 1550. For the foreground elements 1500,1510, two or more of the subframes 1560A, 1560B, 1560C can be combined,without using the masks. Instead, to combine the subframes, theprocessor can average two or more of the subframes 1560A, 1560B, 1560Ctogether to calculate a motion blurred output image 1570. The outputimages 1550 and 1570 can be the same image, or can be two differentimages, which are then combined to obtain the final output image 1580.

FIG. 16 is a flowchart of a method to increase resolution of a displayin postprocessing. In step 1600, a hardware or software processorexecuting instructions described in this application can obtain multipleimages presented on a display by determining a region of the multipleimages visible to a light receiver. To obtain the multiple images, inone implementation, the processor can obtain multiple images recorded bythe camera. In another implementation, the processor can use knowledgeof the camera position, the display position, the camera orientation,and the images presented on the display to determine a portion of animage on the display wall visible to the camera. The camera can be thelight receiver, as described in this application.

The display can present the multiple images at a first frame rate higherthan a frame rate needed to form a perception of motion. The first framerate can be 48 FPS, 96 FPS, 120 FPS, 192 FPS. The first frame rate canbe the number of subframes times 24 FPS, or the number of subframestimes 30.

The light receiver can operate a second frame rate proportional to thefirst frame rate. The second frame rate can be the same as the firstframe rate. Both the first frame rate and the second frame rate arehigher than the frame rate needed to form a perception of motion inproportion to a number of multiple images, e.g., subframes, presented onthe display. For example, if there are three subframes presented foreach frame, the light receiver and the display can operate at 72 FPS,because 72 FPS is equal to 24 FPS×3 subframes. The first and the secondframe rate can be an integer multiple of 24, 25, 30, 24000/1001, or30000/1001 FPS.

In one implementation, to determine the region of the multiple imagesvisible to the light receiver, the processor can record the multipleimages presented on the display using the light receiver. The processorcan register the recorded images among themselves, by identifying pixelsin each image corresponding to pixels in the rest of the images. Thecorrespondence can be 1-to-1, and the upper-left corner pixel in eachimage can line up. The processor can then combine the multiple imagesusing masks to obtain a super resolution output image.

In another implementation, to determine the region of the multipleimages visible to the light receiver, the processor can obtain anindication of a particular time, a position associated with the lightreceiver at the particular time, and a position of the display at theparticular time. For each pixel associated with the light receiver, theprocessor can, based on the indication of the particular time, obtain animage among the multiple images presented on the display. The processorcan, based on the light position associated with the light receiver, andthe position of the display, determine the region associated with theimage that corresponds to each pixel associated with the light receiver.The processor can obtain a color associated with the region associatedwith the image. A region can include multiple pixels in the input image.In other words, one pixel in the output image can come from multiplepixels and the input image, particularly when there is motion blur. Whenthere is motion blur, a pixel in the output image corresponds to a lineacross the screen. To handle that, the output value may have to comefrom multiple subframes fused together, possibly from slightly differentpixel locations.

In step 1610, the processor can obtain a mask corresponding to one ormore images among the multiple images, where the mask indicates aportion of the one or more images among the multiple images to includein an output image. The mask can numerically indicate the portion ofeach image among the multiple images to include in an output image. Forexample, the mask can be a fraction such as one-half, one-third,one-fourth, one-fifth, etc., indicating that one-half, one-third,one-fourth, one-fifth, etc., respectively, of each pixel is visible ateach subframe. The mass can graphically indicate the portion of eachimage among the multiple images to at least partially included in theoutput image, as shown in FIGS. 14A-14C.

In step 1620, the processor can increase resolution of the display inproportion to a number of multiple images presented to the display bycombining, based on the mask, the one or more images among the multipleimages to obtain the output image. To combine the images, the processorcan, based on the mask, determine a first region of a first image amongthe multiple images and a second region of a second image among themultiple images. The processor can combine, by for example adding, thefirst region and the second region to the output image. The mask caninclude an alpha channel indicating a portion of the first region and aportion of the second region to include in the output image. In thatcase, the processor can combine the first region and the second regionto obtain the output image by interpolating between the first region andthe second region based on the mask.

The processor can separate the foreground and the background elementsfrom the recorded image and apply different postprocessing techniques tothe foreground and the background, as explained in FIG. 15 . Theprocessor can identify in the output image a representation of an objectand an image among the multiple images presented on the display. Theobject can be a prop or an actor. The object is not part of the imagepresented on the display. The object and can be disposed between thedisplay and the light receiver. For a region in the output imagebelonging to the representation of the object, the processor can obtaina corresponding region in at least a subset of images among the multipleimages, e.g., subframes, to obtain multiple corresponding regions. Theprocessor can combine the multiple corresponding regions to obtain acombined region. To combine the multiple corresponding regions in thesubframes, the processor can average all the subframes or a subset ofthe subframes, such as two out of the four subframes. The averaging ofthe subframes helps create an appropriate motion blur and reduces theimage noise that is introduced by running the light receiver at a fasterframe rate. The processor can replace the representation of the objectwith the combined region.

To identify in the output image the representation of the object, theprocessor can obtain a first image recorded by the light receiver. Theprocessor can obtain at least a portion of a second image presented onthe display and within a field of view of the light receiver. Theprocessor can determine whether a first region of the first imagematches a second region of the at least a portion of the second image.The region can include one or more pixels. Upon determining that thefirst region and the second region do not match, the processor canidentify the first region as at least a portion of the representation ofthe object.

Visual Content Generation System

FIG. 17 illustrates an example visual content generation system 1700 asmight be used to generate imagery in the form of still images and/orvideo sequences of images. Visual content generation system 1700 mightgenerate imagery of live action scenes, computer-generated scenes, or acombination thereof. In a practical system, users are provided withtools that allow them to specify, at high levels and low levels wherenecessary, what is to go into that imagery. For example, a user might bean animation artist who uses visual content generation system 1700 tocapture interaction between two human actors performing live on a soundstage. The animation artist might replace one of the human actors with acomputer-generated anthropomorphic non-human being that behaves in waysthat mimic the replaced human actor's movements and mannerisms, and thenadd a third character and background scene elements that are allcomputer-generated, in order to tell a desired story or generate desiredimagery.

Still images that are output by visual content generation system 1700might be represented in computer memory as pixel arrays, such as atwo-dimensional array of pixel color values, each associated with apixel having a position in a two-dimensional image array. Pixel colorvalues might be represented by three or more (or fewer) color values perpixel, such as a red value, a green value, and a blue value (e.g., inRGB format). Dimensions of such a two-dimensional array of pixel colorvalues might correspond to a preferred and/or standard display scheme,such as 1920-pixel columns by 1280-pixel rows or 4096-pixel columns by2160-pixel rows, or some other resolution. Images might or might not bestored in a certain structured format, but either way, a desired imagemay be represented as a two-dimensional array of pixel color values. Inanother variation, images are represented by a pair of stereo images forthree-dimensional presentations and in other variations, an imageoutput, or a portion thereof, might represent three-dimensional imageryinstead of just two-dimensional views. In yet other embodiments, pixelvalues are data structures and a pixel value can be associated with apixel and can be a scalar value, a vector, or another data structureassociated with a corresponding pixel. That pixel value might includecolor values, or not, and might include depth values, alpha values,weight values, object identifiers, or other pixel value components.

A stored video sequence might include a plurality of images such as thestill images described above, but where each image of the plurality ofimages has a place in a timing sequence and the stored video sequence isarranged so that when each image is displayed in order, at a timeindicated by the timing sequence, the display presents what appears tobe moving and/or changing imagery. In one representation, each image ofthe plurality of images is a video frame having a specified frame numberthat corresponds to an amount of time that would elapse from when avideo sequence begins playing until that specified frame is displayed. Aframe rate might be used to describe how many frames of the stored videosequence are displayed per unit time. Example video sequences mightinclude 24 FPS, 50 FPS, 140 FPS, or other frame rates. In someembodiments, frames are interlaced or otherwise presented for display,but for clarity of description, in some examples, it is assumed that avideo frame has one specified display time. Other variations might becontemplated, however.

One method of creating a video sequence is to simply use a video camerato record a live action scene, i.e., events that physically occur andcan be recorded by a video camera. The events being recorded can beevents to be interpreted as viewed (such as seeing two human actors talkto each other) and/or can include events to be interpreted differentlydue to clever camera operations (such as moving actors about a stage tomake one appear larger than the other despite the actors actually beingof similar build, or using miniature objects with other miniatureobjects so as to be interpreted as a scene containing life-sizedobjects).

Creating video sequences for storytelling or other purposes often callsfor scenes that cannot be created with live actors, such as a talkingtree, an anthropomorphic object, space battles, and the like. Such videosequences might be generated computationally rather than capturing lightfrom live scenes. In some instances, an entirety of a video sequencemight be generated computationally, as in the case of acomputer-animated feature film. In some video sequences, it is desirableto have some computer-generated imagery and some live action, perhapswith some careful merging of the two.

While computer-generated imagery might be creatable by manuallyspecifying each color value for each pixel in each frame, this is likelytoo tedious to be practical. As a result, a creator uses various toolsto specify the imagery at a higher level. As an example, an artist mightspecify the positions in a scene space, such as a three-dimensionalcoordinate system, of objects and/or lighting, as well as a cameraviewpoint, and a camera view plane. From that, a rendering engine couldtake all of those as inputs, and compute each of the pixel color valuesin each of the frames. In another example, an artist might specifyposition and movement of an articulated object having some specifiedtexture rather than specifying the color of each pixel representing thatarticulated object in each frame.

In a specific example, a rendering engine performs ray tracing wherein apixel color value is determined by computing which objects lie along aray traced in the scene space from the camera viewpoint through a pointor portion of the camera view plane that corresponds to that pixel. Forexample, a camera view plane might be represented as a rectangle havinga position in the scene space that is divided into a grid correspondingto the pixels of the ultimate image to be generated, and if a raydefined by the camera viewpoint in the scene space and a given pixel inthat grid first intersects a solid, opaque, blue object, that givenpixel is assigned the color blue. Of course, for moderncomputer-generated imagery, determining pixel colors—and therebygenerating imagery—can be more complicated, as there are lightingissues, reflections, interpolations, and other considerations.

As illustrated in FIG. 17 , a live action capture system 1702 captures alive scene that plays out on a stage 1704. Live action capture system1702 is described herein in greater detail, but might include computerprocessing capabilities, image processing capabilities, one or moreprocessors, program code storage for storing program instructionsexecutable by the one or more processors, as well as user input devicesand user output devices, not all of which are shown.

In a specific live action capture system, cameras 1706(1) and 1706(2)capture the scene, while in some systems, there might be other sensor(s)1708 that capture information from the live scene (e.g., infraredcameras, infrared sensors, motion capture (“mo-cap”) detectors, etc.).On stage 1704, there might be human actors, animal actors, inanimateobjects, background objects, and possibly an object such as a greenscreen 1710 that is designed to be captured in a live scene recording insuch a way that it is easily overlaid with computer-generated imagery.Stage 1704 might also contain objects that serve as fiducials, such asfiducials 1712(1)-(3), that might be used post-capture to determinewhere an object was during capture. A live action scene might beilluminated by one or more lights, such as an overhead light 1714.

During or following the capture of a live action scene, live actioncapture system 1702 might output live action footage to a live actionfootage storage 1720. A live action processing system 1722 might processlive action footage to generate data about that live action footage andstore that data into a live action metadata storage 1724. Live actionprocessing system 1722 might include computer processing capabilities,image processing capabilities, one or more processors, program codestorage for storing program instructions executable by the one or moreprocessors, as well as user input devices and user output devices, notall of which are shown. Live action processing system 1722 might processlive action footage to determine boundaries of objects in a frame ormultiple frames, determine locations of objects in a live action scene,where a camera was relative to some action, distances between movingobjects and fiducials, etc. Where elements have sensors attached to themor are detected, the metadata might include location, color, andintensity of overhead light 1714, as that might be useful inpost-processing to match computer-generated lighting on objects that arecomputer-generated and overlaid on the live action footage. Live actionprocessing system 1722 might operate autonomously, perhaps based onpredetermined program instructions, to generate and output the liveaction metadata upon receiving and inputting the live action footage.The live action footage can be camera-captured data as well as data fromother sensors.

An animation creation system 1730 is another part of visual contentgeneration system 1700. Animation creation system 1730 might includecomputer processing capabilities, image processing capabilities, one ormore processors, program code storage for storing program instructionsexecutable by the one or more processors, as well as user input devicesand user output devices, not all of which are shown. Animation creationsystem 1730 might be used by animation artists, managers, and others tospecify details, perhaps programmatically and/or interactively, ofimagery to be generated. From user input and data from a database orother data source, indicated as a data store 1732, animation creationsystem 1730 might generate and output data representing objects (e.g., ahorse, a human, a ball, a teapot, a cloud, a light source, a texture,etc.) to an object storage 1734, generate and output data representing ascene into a scene description storage 1736, and/or generate and outputdata representing animation sequences to an animation sequence storage1738.

Scene data might indicate locations of objects and other visualelements, values of their parameters, lighting, camera location, cameraview plane, and other details that a rendering engine 1750 might use torender computer-generated imagery (CGI). For example, scene data mightinclude the locations of several articulated characters, backgroundobjects, lighting, etc., specified in a two-dimensional space,three-dimensional space, or other dimensional space (such as a2.5-dimensional space, three-quarter dimensions, pseudo-3D spaces, etc.)along with locations of a camera viewpoint and view plane from which torender imagery. For example, scene data might indicate that there is tobe a red, fuzzy, talking dog in the right half of a video and astationary tree in the left half of the video, all illuminated by abright point light source that is above and behind the camera viewpoint.In some cases, the camera viewpoint is not explicit, but can bedetermined from a viewing frustum. In the case of imagery that is to berendered to a rectangular view, the frustum would be a truncatedpyramid. Other shapes for a rendered view are possible and the cameraview plane could be different for different shapes.

Animation creation system 1730 might be interactive, allowing a user toread in animation sequences, scene descriptions, object details, etc.,and edit those, possibly returning them to storage to update or replaceexisting data. As an example, an operator might read in objects fromobject storage into a baking processor 1742 that would transform thoseobjects into simpler forms and return those to object storage 1734 asnew or different objects. For example, an operator might read in anobject that has dozens of specified parameters (movable joints, coloroptions, textures, etc.), select some values for those parameters, andthen save a baked object that is a simplified object with now-fixedvalues for those parameters.

Rather than requiring user specification of each detail of a scene, datafrom data store 1732 might be used to drive object presentation. Forexample, if an artist is creating an animation of a spaceship passingover the surface of the Earth, instead of manually drawing or specifyinga coastline, the artist might specify that animation creation system1730 is to read data from data store 1732 in a file containingcoordinates of Earth coastlines and generate background elements of ascene using that coastline data.

Animation sequence data might be in the form of time series of data forcontrol points of an object that has attributes that are controllable.For example, an object might be a humanoid character with limbs andjoints that are movable in manners similar to typical human movements.An artist can specify an animation sequence at a high level, such as“the left hand moves from location (X1, Y1, Z1) to (X2, Y2, Z2) overtime T1 to T2,” at a lower level (e.g., “move the elbow joint 2.5degrees per frame”), or even at a very high level (e.g., “character Ashould move, consistent with the laws of physics that are given for thisscene, from point P1 to point P2 along a specified path”).

Animation sequences in an animated scene might be specified by whathappens in a live action scene. An animation driver generator 1744 mightread in live action metadata, such as data representing movements andpositions of body parts of a live actor during a live action scene.Animation driver generator 1744 might generate corresponding animationparameters to be stored in animation sequence storage 1738 for use inanimating a CGI object. This can be useful where a live action scene ofa human actor is captured while wearing mo-cap fiducials (e.g.,high-contrast markers outside actor clothing, high-visibility paint onactor skin, face, etc.) and the movement of those fiducials isdetermined by live action processing system 1722. Animation drivergenerator 1744 might convert that movement data into specifications ofhow joints of an articulated CGI character are to move over time.

A rendering engine 1750 can read in animation sequences, scenedescriptions, and object details, as well as rendering engine controlinputs, such as a resolution selection and a set of renderingparameters. Resolution selection might be useful for an operator tocontrol a trade-off between speed of rendering and clarity of detail, asspeed might be more important than clarity for a movie maker to testsome interaction or direction, while clarity might be more importantthan speed for a movie maker to generate data that will be used forfinal prints of feature films to be distributed. Rendering engine 1750might include computer processing capabilities, image processingcapabilities, one or more processors, program code storage for storingprogram instructions executable by the one or more processors, as wellas user input devices and user output devices, not all of which areshown.

Visual content generation system 1700 can also include a merging system1760 that merges live footage with animated content. The live footagemight be obtained and input by reading from live action footage storage1720 to obtain live action footage, by reading from live action metadatastorage 1724 to obtain details such as presumed segmentation in capturedimages segmenting objects in a live action scene from their background(perhaps aided by the fact that green screen 1710 was part of the liveaction scene), and by obtaining CGI imagery from rendering engine 1750.

A merging system 1760 might also read data from rulesets formerging/combining storage 1762. A very simple example of a rule in aruleset might be “obtain a full image including a two-dimensional pixelarray from live footage, obtain a full image including a two-dimensionalpixel array from rendering engine 1750, and output an image where eachpixel is a corresponding pixel from rendering engine 1750 when thecorresponding pixel in the live footage is a specific color of green,otherwise output a pixel value from the corresponding pixel in the livefootage.”

Merging system 1760 might include computer processing capabilities,image processing capabilities, one or more processors, program codestorage for storing program instructions executable by the one or moreprocessors, as well as user input devices and user output devices, notall of which are shown. Merging system 1760 might operate autonomously,following programming instructions, or might have a user interface orprogrammatic interface over which an operator can control a mergingprocess. In some embodiments, an operator can specify parameter valuesto use in a merging process and/or might specify specific tweaks to bemade to an output of merging system 1760, such as modifying boundariesof segmented objects, inserting blurs to smooth out imperfections, oradding other effects. Based on its inputs, merging system 1760 canoutput an image to be stored in a static image storage 1770 and/or asequence of images in the form of video to be stored in ananimated/combined video storage 1772.

Thus, as described, visual content generation system 1700 can be used togenerate video that combines live action with computer-generatedanimation using various components and tools, some of which aredescribed in more detail herein. While visual content generation system1700 might be useful for such combinations, with suitable settings, itcan be used for outputting entirely live action footage or entirely CGIsequences. The code may also be provided and/or carried by a transitorycomputer-readable medium, e.g., a transmission medium such as in theform of a signal transmitted over a network.

According to one embodiment, the techniques described herein areimplemented by one or more generalized computing systems programmed toperform the techniques pursuant to program instructions in firmware,memory, other storage, or a combination. Special-purpose computingdevices may be used, such as desktop computer systems, portable computersystems, handheld devices, networking devices or any other device thatincorporates hard-wired and/or program logic to implement thetechniques.

One embodiment might include a carrier medium carrying image data orother data having details generated using the methods described herein.The carrier medium can comprise any medium suitable for carrying theimage data or other data, including a storage medium, e.g., solid-statememory, an optical disk or a magnetic disk, or a transient medium, e.g.,a signal carrying the image data such as a signal transmitted over anetwork, a digital signal, a radio frequency signal, an acoustic signal,an optical signal, or an electrical signal.

Computer System

FIG. 18 is a block diagram that illustrates a computer system 1800 uponwhich the computer systems of the systems described herein and/or visualcontent generation system 1700 (see FIG. 17 ) may be implemented.Computer system 1800 includes a bus 1802 or other communicationmechanism for communicating information, and a processor 1804 coupledwith bus 1802 for processing information. Processor 1804 may be, forexample, a general-purpose microprocessor.

Computer system 1800 also includes a main memory 1806, such as arandom-access memory (RAM) or other dynamic storage device, coupled tobus 1802 for storing information and instructions to be executed byprocessor 1804. Main memory 1806 may also be used for storing temporaryvariables or other intermediate information during execution ofinstructions to be executed by processor 1804. Such instructions, whenstored in non-transitory storage media accessible to processor 1804,render computer system 1800 into a special-purpose machine that iscustomized to perform the operations specified in the instructions.

Computer system 1800 further includes a read only memory (ROM) 1808 orother static storage device coupled to bus 1802 for storing staticinformation and instructions for processor 1804. A storage device 1810,such as a magnetic disk or optical disk, is provided and coupled to bus1802 for storing information and instructions.

Computer system 1800 may be coupled via bus 1802 to a display 1812, suchas a computer monitor, for displaying information to a computer user. Aninput device 1814, including alphanumeric and other keys, is coupled tobus 1802 for communicating information and command selections toprocessor 1804. Another type of user input device is a cursor control1816, such as a mouse, a trackball, or cursor direction keys forcommunicating direction information and command selections to processor1804 and for controlling cursor movement on display 1812. This inputdevice typically has two degrees of freedom in two axes, a first axis(e.g., x) and a second axis (e.g., y), that allows the device to specifypositions in a plane.

Computer system 1800 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 1800 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 1800 in response to processor 1804 executing one or moresequences of one or more instructions contained in main memory 1806.Such instructions may be read into main memory 1806 from another storagemedium, such as storage device 1810. Execution of the sequences ofinstructions contained in main memory 1806 causes processor 1804 toperform the process steps described herein. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions.

The term “storage media” as used herein refers to any non-transitorymedia that stores data and/or instructions that cause a machine tooperate in a specific fashion. Such storage media may includenon-volatile media and/or volatile media. Non-volatile media includes,for example, optical or magnetic disks, such as storage device 1810.Volatile media includes dynamic memory, such as main memory 1806. Commonforms of storage media include, for example, a floppy disk, a flexibledisk, a hard disk, a solid state drive, magnetic tape, or any othermagnetic data storage medium, a CD-ROM, any other optical data storagemedium, any physical medium with patterns of holes, a RAM, a PROM, anEPROM, a FLASH-EPROM, NVRAM, or any other memory chip or cartridge.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire, and fiber optics, including thewires that include bus 1802. Transmission media can also take the formof acoustic or light waves, such as those generated during radio-waveand infrared data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 1804 for execution. Forexample, the instructions may initially be carried on a magnetic disk orsolid-state drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over anetwork connection. A modem or network interface local to computersystem 1800 can receive the data. Bus 1802 carries the data to mainmemory 1806, from which processor 1804 retrieves and executes theinstructions. The instructions received by main memory 1806 mayoptionally be stored on storage device 1810 either before or afterexecution by processor 1804.

Computer system 1800 also includes a communication interface 1818coupled to bus 1802. Communication interface 1818 provides a two-waydata communication coupling to a network link 1820 that is connected toa local network 1822. For example, communication interface 1818 may be anetwork card, a modem, a cable modem, or a satellite modem to provide adata communication connection to a corresponding type of telephone lineor communications line. Wireless links may also be implemented. In anysuch implementation, communication interface 1818 sends and receiveselectrical, electromagnetic, or optical signals that carry digital datastreams representing various types of information.

Network link 1820 typically provides data communication through one ormore networks to other data devices. For example, network link 1820 mayprovide a connection through local network 1822 to a host computer 1824or to data equipment operated by an Internet Service Provider (ISP)1826. ISP 1826 in turn provides data communication services through theworldwide packet data communication network now commonly referred to asthe “Internet” 1828. Local network 1822 and Internet 1828 both useelectrical, electromagnetic, or optical signals that carry digital datastreams. The signals through the various networks and the signals onnetwork link 1820 and through communication interface 1818, which carrythe digital data to and from computer system 1800, are example forms oftransmission media.

Computer system 1800 can send messages and receive data, includingprogram code, through the network(s), network link 1820, andcommunication interface 1818. In the Internet example, a server 1830might transmit a requested code for an application program through theInternet 1828, ISP 1826, local network 1822, and communication interface1818. The received code may be executed by processor 1804 as it isreceived, and/or stored in storage device 1810, or in other non-volatilestorage for later execution.

Operations of processes described herein can be performed in anysuitable order unless otherwise indicated herein or otherwise clearlycontradicted by context. Processes described herein (or variationsand/or combinations thereof) may be performed under the control of oneor more computer systems configured with executable instructions and maybe implemented as code (e.g., executable instructions, one or morecomputer programs, or one or more applications) executing collectivelyon one or more processors, by hardware or combinations thereof. The codemay be stored on a computer-readable storage medium, for example, in theform of a computer program comprising a plurality of instructionsexecutable by one or more processors. The computer-readable storagemedium may be non-transitory. The code may also be provided and/orcarried by a transitory computer-readable medium, e.g., a transmissionmedium such as in the form of a signal transmitted over a network.

Conjunctive language, such as phrases of the form “at least one of A, B,and C,” or “at least one of A, B and C,” unless specifically statedotherwise or otherwise clearly contradicted by context, is otherwiseunderstood within the context as used in general to present that anitem, term, etc., may be either A or B or C, or any nonempty subset ofthe set of A and B and C. For instance, in the illustrative example of aset having three members, the conjunctive phrases “at least one of A, B,and C” and “at least one of A, B and C” refer to any of the followingsets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, suchconjunctive language is not generally intended to imply that certainembodiments require at least one of A, at least one of B, and at leastone of C each to be present.

The use of examples, or exemplary language (e.g., “such as”) providedherein, is intended merely to better illuminate embodiments of theinvention and does not pose a limitation on the scope of the inventionunless otherwise claimed. No language in the specification should beconstrued as indicating any non-claimed element as essential to thepractice of the invention.

In the foregoing specification, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. The specification and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense. The sole and exclusive indicator of the scope of the invention,and what is intended by the applicants to be the scope of the invention,is the literal and equivalent scope of the set of claims that issue fromthis application, in the specific form in which such claims issue,including any subsequent correction.

Further embodiments can be envisioned to one of ordinary skill in theart after reading this disclosure. In other embodiments, combinations orsub-combinations of the above-disclosed invention can be advantageouslymade. The example arrangements of components are shown for purposes ofillustration and combinations, additions, re-arrangements, and the likeare contemplated in alternative embodiments of the present invention.Thus, while the invention has been described with respect to exemplaryembodiments, one skilled in the art will recognize that numerousmodifications are possible.

For example, the processes described herein may be implemented usinghardware components, software components, and/or any combinationthereof. The specification and drawings are, accordingly, to be regardedin an illustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the invention asset forth in the claims and that the invention is intended to cover allmodifications and equivalents within the scope of the following claims.

All references, including publications, patent applications, andpatents, cited herein are hereby incorporated by reference to the sameextent as if each reference were individually and specifically indicatedto be incorporated by reference and were set forth in its entiretyherein.

REMARKS

The terms “example,” “embodiment,” and “implementation” are usedinterchangeably. For example, references to “one example” or “anexample” in the disclosure can be, but not necessarily are, referencesto the same implementation; and, such references mean at least one ofthe implementations. The appearances of the phrase “in one example” arenot necessarily all referring to the same example, nor are separate oralternative examples mutually exclusive of other examples. A feature,structure, or characteristic described in connection with an example canbe included in another example of the disclosure. Moreover, variousfeatures are described which can be exhibited by some examples and notby others. Similarly, various requirements are described which can berequirements for some examples but no other examples.

The terminology used herein should be interpreted in its broadestreasonable manner, even though it is being used in conjunction withcertain specific examples of the invention. The terms used in thedisclosure generally have their ordinary meanings in the relevanttechnical art, within the context of the disclosure, and in the specificcontext where each term is used. A recital of alternative language orsynonyms does not exclude the use of other synonyms. Specialsignificance should not be placed upon whether or not a term iselaborated or discussed herein. The use of highlighting has no influenceon the scope and meaning of a term. Further, it will be appreciated thatthe same thing can be said in more than one way.

Unless the context clearly requires otherwise, throughout thedescription and the claims, the words “comprise,” “comprising,” and thelike are to be construed in an inclusive sense, as opposed to anexclusive or exhaustive sense; that is to say, in the sense of“including, but not limited to.” As used herein, the terms “connected,”“coupled,” or any variant thereof means any connection or coupling,either direct or indirect, between two or more elements; the coupling orconnection between the elements can be physical, logical, or acombination thereof. Additionally, the words “herein,” “above,” “below,”and words of similar import can refer to this application as a whole andnot to any particular portions of this application. Where contextpermits, words in the above Detailed Description using the singular orplural number may also include the plural or singular number,respectively. The word “or” in reference to a list of two or more itemscovers all of the following interpretations of the word: any of theitems in the list, all of the items in the list, and any combination ofthe items in the list. The term “module” refers broadly to softwarecomponents, firmware components, and/or hardware components.

While specific examples of technology are described above forillustrative purposes, various equivalent modifications are possiblewithin the scope of the invention, as those skilled in the relevant artwill recognize. For example, while processes or blocks are presented ina given order, alternative implementations can perform routines havingsteps, or employ systems having blocks, in a different order, and someprocesses or blocks may be deleted, moved, added, subdivided, combined,and/or modified to provide alternative or sub-combinations. Each ofthese processes or blocks can be implemented in a variety of differentways. Also, while processes or blocks are at times shown as beingperformed in series, these processes or blocks can instead be performedor implemented in parallel, or can be performed at different times.Further, any specific numbers noted herein are only examples such thatalternative implementations can employ differing values or ranges.

Details of the disclosed implementations can vary considerably inspecific implementations while still being encompassed by the disclosedteachings. As noted above, particular terminology used when describingfeatures or aspects of the invention should not be taken to imply thatthe terminology is being redefined herein to be restricted to anyspecific characteristics, features, or aspects of the invention withwhich that terminology is associated. In general, the terms used in thefollowing claims should not be construed to limit the invention to thespecific examples disclosed herein, unless the above DetailedDescription explicitly defines such terms. Accordingly, the actual scopeof the invention encompasses not only the disclosed examples, but alsoall equivalent ways of practicing or implementing the invention underthe claims. Some alternative implementations can include additionalelements to those implementations described above or include fewerelements.

Any patents and applications and other references noted above, and anythat may be listed in accompanying filing papers, are incorporatedherein by reference in their entireties, except for any subject matterdisclaimers or disavowals, and except to the extent that theincorporated material is inconsistent with the express disclosureherein, in which case the language in this disclosure controls. Aspectsof the invention can be modified to employ the systems, functions, andconcepts of the various references described above to provide yetfurther implementations of the invention.

To reduce the number of claims, certain implementations are presentedbelow in certain claim forms, but the applicant contemplates variousaspects of an invention in other forms. For example, aspects of a claimcan BE recited in a means-plus-function form or in other forms, such asbeing embodied in a computer-readable medium. A claim intended to beinterpreted as a means-plus-function claim will use the words “meansfor.” However, the use of the term “for” in any other context is notintended to invoke a similar interpretation. The applicant reserves theright to pursue such additional claim forms in either this applicationor in a continuing application.

We claim:
 1. A method comprising: presenting multiple images to a lightreceiver via a display, wherein the display is configured to present themultiple images at a first frame rate, wherein the light receiver isconfigured to operate at a second frame rate proportional to the firstframe rate; providing a mask corresponding to one or more images amongthe multiple images, wherein the mask indicates a portion of the one ormore images among the multiple images to include in an output image; andcausing an increase in an apparent resolution of the display inproportion to a number of multiple images presented on the display bycausing a combination, based on the mask, of the one or more imagesamong the multiple images to obtain the output image.
 2. The method ofclaim 1, comprising: creating an impression of a motion blur by:providing an indication of a particular time, a position associated withthe light receiver at the particular time, and a position of the displayat the particular time; based on the indication of the particular time,providing one or more images among the multiple images presented on thedisplay; based on the position associated with the light receiver, andthe position of the display, causing a determination of one or moreregions associated with the one or more images that corresponds to apixel associated with the light receiver; providing one or more colorsof the one or more regions associated with the one or more images; andcausing a combination of the one or more colors of the one or moreregions associated with the one or more images to obtain the impressionof the motion blur.
 3. The method of claim 1, comprising: causing anidentification in the output image of a representation of an object andan image among the multiple images presented on the display, wherein theobject is not a part of the image presented on the display; for a regionin the output image belonging to the representation of the objectcausing an obtaining of a corresponding region in at least a subset ofimages among the multiple images to obtain multiple correspondingregions; causing a combining of the multiple corresponding regions toobtain a combined region; and causing a replacement of therepresentation of the object with the combined region.
 4. The method ofclaim 3, wherein causing the identification in the output image of therepresentation of the object comprises: causing an obtaining of a firstimage recorded by the light receiver; causing an obtaining of at least aportion of a second image presented on the display and within a field ofview of the light receiver; causing a determination of whether a firstregion of the first image corresponds to a second region of the portionof the second image; and upon determining that the first region and thesecond region do not correspond, causing the identification of the firstregion as at least a portion of the representation of the object.
 5. Themethod of claim 1, wherein causing the combination of the one or moreimages comprises: based on the mask, cause a determination of a firstregion of a first image among the multiple images to include in theoutput image and a second region of a second image among the multipleimages to include in the output image; and cause the combination of thefirst region and the second region to obtain the output image.
 6. Asystem comprising: at least one hardware processor; and at least onenon-transitory memory storing instructions, which, when executed by theat least one hardware processor, cause the system to: present multipleimages to a light receiver via a display, wherein the display isconfigured to present the multiple images at a first frame rate, whereinthe light receiver is configured to operate at a second frame rateproportional to the first frame rate; providing a mask corresponding toone or more images among the multiple images, wherein the mask indicatesa portion of the one or more images among the multiple images to includein an output image; and cause an increase in an apparent resolution ofthe display in proportion to a number of multiple images presented onthe display by causing a combination, based on the mask, of the one ormore images among the multiple images to obtain the output image.
 7. Thesystem of claim 6, comprising the instructions to: provide an indicationof a particular time, a position associated with the light receiver atthe particular time, and a position of the display at the particulartime; based on the indication of the particular time, provide an imageamong the multiple images presented on the display; based on theposition associated with the light receiver, and the position of thedisplay, cause a determination of a region associated with the imagethat corresponds to a pixel associated with the light receiver; andprovide a color associated with the region associated with the image. 8.The system of claim 6, comprising instructions to: cause identificationin the output image of a representation of an object and an image amongthe multiple images presented on the display, wherein the object is nota part of the image presented on the display; for a region in the outputimage belonging to the representation of the object cause an obtainingof a corresponding region in at least a subset of images among themultiple images to obtain multiple corresponding regions; cause acombination of the multiple corresponding regions to obtain a combinedregion; and cause a replacement of the representation of the object withthe combined region.
 9. The system of claim 8, the instructions to causean identification in the output image of the representation of theobject comprising instructions to: cause an obtaining of a first imagerecorded by the light receiver; cause an obtaining of at least a portionof a second image presented on the display and within a field of view ofthe light receiver; cause a determination of whether a first region ofthe first image corresponds to a second region of the portion of thesecond image; and upon determining that the first region and the secondregion do not correspond, cause an identification of the first region asat least a portion of the representation of the object.
 10. The systemof claim 6, wherein the instructions to provide the mask compriseinstructions to: provide multiple masks, wherein each mask among themultiple masks graphically indicates the portion of the one or moreimages among the multiple images to at least partially included in theoutput image.
 11. The system of claim 6, wherein the instructions tocause the combination of the one or more images comprise instructionsto: based on the mask, cause a determination of a first region of afirst image among the multiple images and a second region of a secondimage among the multiple images; and cause the combination of the firstregion and the second region to obtain the output image.
 12. The systemof claim 6, wherein the instructions to cause the combination of the oneor more images comprise instructions to: based on the mask, cause adetermination of a first region of a first image among the multipleimages to include in the output image and a second region of a secondimage among the multiple images to include in the output image; andcause the combination of the first region and the second region toobtain the output image by interpolating, based on the mask, between thefirst region and the second region.
 13. At least one computer-readablestorage medium, excluding transitory signals and carrying instructions,which, when executed by at least one data processor of a system, causesthe system to: present multiple images to a light receiver via adisplay, wherein the display is configured to present the multipleimages at a first frame rate, wherein the light receiver is configuredto operate at a second frame rate proportional to the first frame rate;providing a mask corresponding to one or more images among the multipleimages, wherein the mask indicates a portion of the one or more imagesamong the multiple images to include in an output image; and cause anincrease in an apparent resolution of the display in proportion to anumber of multiple images presented on the display by causing acombination, based on the mask, of the one or more images among themultiple images to obtain the output image.
 14. The at least onecomputer-readable storage medium of claim 13, comprising theinstructions to: provide an indication of a particular time, a positionassociated with the light receiver at the particular time, and aposition of the display at the particular time; for each pixelassociated with the light receiver perform steps including: based on theindication of the particular time, provide an image among the multipleimages; based on the position associated with the light receiver, andthe position of the display, cause a determination of a regionassociated with the image that corresponds to a pixel associated withthe light receiver; and cause obtaining of a color associated with theregion associated with the image.
 15. The at least one computer-readablestorage medium of claim 13, comprising instructions to: cause anidentification in the output image of a representation of an object andan image among the multiple images presented on the display, wherein theobject is not a part of the image presented on the display; for a regionin the output image belonging to the representation of the object causeobtaining of a corresponding region in at least a subset of images amongthe multiple images to obtain multiple corresponding regions; cause acombining of the multiple corresponding regions to obtain a combinedregion; and cause a replacing of the representation of the object withthe combined region.
 16. The at least one computer-readable storagemedium of claim 15, the instructions to cause the identification in theoutput image of the representation of the object comprising instructionsto: cause obtaining of a first image recorded by the light receiver;cause obtaining at least a portion of a second image presented on thedisplay and within a field of view of the light receiver; causedetermination of whether a first region of the first image correspondsto a second region of the at least the portion of the second image; andupon determining that the first region and the second region do notcorrespond, cause identifying of the first region as at least a portionof the representation of the object.
 17. The at least onecomputer-readable storage medium of claim 13, wherein the instructionsto provide the mask comprise instructions to: provide the masknumerically indicating the portion of the one or more images among themultiple images to include in the output image.
 18. The at least onecomputer-readable storage medium of claim 13, wherein the instructionsto cause a combination of the one or more images comprise instructionsto: based on the mask cause a determination of a first region of a firstimage among the multiple images and a second region of a second imageamong the multiple images; and cause a combination of the first regionand the second region to obtain the output image.
 19. The at least onecomputer-readable storage medium of claim 13, wherein the instructionsto cause a combination of the one or more images comprise instructionsto: based on the mask, cause a determination of a first region of afirst image among the multiple images to include in the output image anda second region of a second image among the multiple images to includein the output image; and cause a combination of the first region and thesecond region to obtain the output image by interpolating, based on themask, between the first region and the second region.
 20. The at leastone computer-readable storage medium of claim 13, comprisinginstructions to: create an impression of a motion blur by: providing anindication of a particular time, a position associated with the lightreceiver at the particular time, and a position of the display at theparticular time; based on the indication of the particular time,providing one or more images among the multiple images presented on thedisplay; based on the position associated with the light receiver, andthe position of the display, causing a determination of one or moreregions associated with the one or more images that corresponds to apixel associated with the light receiver; providing one or more colorsof the one or more regions associated with the one or more images; andcausing a combination of the one or more colors of the one or moreregions associated with the one or more images to obtain the impressionof the motion blur.